New rules and jmapquery

[n5bb]

Originally Posted by xyzzy

Finally got a response on my ticket on why I don't ever see jmapquery's. It's because it never occurred to me different sieve code for conditions would be generated when you create a rule using no-preview mode as opposed to the new preview mode. I wasn't using preview mode in all my testing since I dislike its UI. But as it turns out creating a rule in preview mode causes the jmapquery to be generated. I suppose checking, say for Subject, with a jmapquery is the same as using a header :contains created with no preview mode. Oh well, it is what it is.

Wow - I never suspected that!

• See example below showing both creation modes available in the current beta UI (no-preview, then standard rules).
• The Help currently uses the term "Sieve constructor" rather than "no-preview".
• To me it's a bit confusing to call the new mode "standard" and the more advanced mode (in most respects) "no-preview mode (regular expressions supported)".
• I think it might be better to call the two rule creation modes "basic (with preview)" and "advanced (without preview)".

 ### 4a. Calculate rule actions
if 
  allof( not string :is "${stop}" "Y",
    header :contains "Subject" "ZBJ"
  )
{
  set "L0_Test" "Y";
}

# Search: "subject:SNEEZE"
if 
  allof( not string :is "${stop}" "Y",
    jmapquery text:
  {
     "subject" : "SNEEZE"
  }
.
  )
{
  set "L0_Test" "Y";
}

[xyzzy]

Originally Posted by n5bb

The Help currently uses the term "Sieve constructor" rather than "no-preview".

Yeah I caught that too going back and forth with the FM tech support guy handling my ticket so I mentioned it.

He also indicated another oddity with jmapquery. First the restriction on the rules conditions are because search can't handle any others with it and the surprising one I didn't know about search is, and I quote...

There are some subtle differences in old and new rule behaviors that will, I think, generally be more like what most users want. An old-style rule like "body contains goose" will not match "geese". A new-style rule will. In effect, new-style rules behave exactly like search, which is why they can be previewed with search.

I'm looking at section 4.4 of RFC8621 and I can't see where it says a mail query does stuff like that. If it does that's not a exact mapping to a header :contains and IMO can be confusing for the user. I view a search as searching for exactly what you are asking for and not interpreting as find stuff it thinks you are asking for.

At any rate it doesn't matter I don't like the UI for constructing rules in that preview mode and always use the "no preview" mode which is why I was never able to cause the jmapquery's to be generated.

[n5bb]

In effect, new-style rules behave exactly like search, which is why they can be previewed with search.

I don't understand the "goose" vs "geese" example. I have many hits for both goose and geese in my email.

• When I make a simple search for goose in the search box I only see goose hits. I see the same thing when I create a rule with preview for goose in the body, which produces a search for body:goose.
• When I make a simple search for geese in the search box I only see geese hits. I see the same thing when I create a rule with preview for geese in the body, which produces a search for body:geese.

Can someone explain the goose search I see in beta mode with labels enabled? Was that a real example which the Fastmail staff member had tested? If so, what am I doing wrong?

Bill

[gardenweed]

Background
I have a bundle of individual rules that filter selected emails into a folder called Subscriptions.
It's a simple Inbox management rule where I dump all those not so important subscription emails so as not to clutter my inbox.

When using sieve years ago, the code for this rule was one block which tested the from header and using ifany (if I recall) dumped the email into the Subscriptions folder. The block of sieve was quite long - ie many subscriptions - but it was all together in one block. Nice and neat.

When the new Rules came along (not the new new rules) this block of sieve got converted to a bucket load of individual rule tests. Looked messy to me but worked. So many rules for one action - file into Subscriptions folder.

Now, with the new-new rules, we can have multiple tests again, ie "If any"

My question:
For my rule that puts email into my Subscriptions bucket - is it better to have many individual rules, or to have one rule that lists all the email from-addresses that should direct to my Subscriptions bucket? Pros/Cons?

[xyzzy]

Originally Posted by n5bb

I don't understand the "goose" vs "geese" example. I have many hits for both goose and geese in my email. - - -

Can someone explain the goose search I see in beta mode with labels enabled? Was that a real example which the Fastmail staff member had tested? If so, what am I doing wrong?

I only quoted what he said. I doubt he tested it since we were just conversing back and forth through the ticket. What I said to him is I didn't see anything in RFC8621 section 4.4 that implied searches were so relaxed and if they are FM better document it that or it will confuse users. Remember, for this ticket at least, I was dealing with a 1st level tech support. I always take 1st level tech support for most any service with a gain of salt! My guess is you aren't doing anything wrong. Who want's a search that assumes what you really want to look for?

Maybe I should go back to that ticket, challenge him on his statement, and escalate it if necessary.

By the way I don't think being in labels or folders mode is significant for this search stuff.

[n5bb]

Since you asked a philosophical question, I will respond with my philosophy on this issue.

• There are several problems with using the From address to sort such messages, including:
  • Many organizations use more than one From address. They may even use more than one From domain.
  • Anyone can spoof a From address. So a good phishing email might not get caught by the spam filter and would be filed into that special folder. It might be hard for you to determine if that message was really from your bank.
• So I use a special subdomain address at an alias I control when I sign up. You can nearly always change the existing address you signed up with to a new one. See: https://www.fastmail.com/help/receive/addressing.html
• You could use a single address for all such subscriptions, but if a spammer discovered that single address, you wouldn't know which organization leaked it.
• So I use a unique subdomain address for each subscription sign-up. For example, let's say you had control over the Fastmail alias "subs@example.org". if you signed up with a bank, newspaper, and gas utility you might use the addresses;
  • bankname@subs.example.org
  • newspaper@subs.example.org
  • gas@subs.example.org
• If your Fastmail main account address was user@example.org, you could then set up the target of the "subs@example.org" Fastmail alias to point to a particular delivery folder. For example, you could point the alias "subs@example.org" to deliver to "user+subscriptions@example.org" if you had the folder "subscriptions" (case is ignored).
• If you wanted to set up a subfolder below "subscriptions" for each of the above examples, messages sent TO those subdomain addresses will be automatically filed as follows:
  • X-Delivered-To: bankname@subs.example.org will result in X-Resolved-To: user+subs.bankname@example.org, which files the message into the subs.bankname folder (bankname subdirectory of subs).
  • X-Delivered-To: newspaper@subs.example.org will result in X-Resolved-To: user+subs.newspaper@example.org, which files the message into the subs.newspaper folder (newspaper subdirectory of subs).
  • X-Delivered-To: gas@subs.example.org will result in X-Resolved-To: user+subs.gas@example.org, which files the message into the subs.gas folder (gas subdirectory of subs).
• If you don't want to create the individual subfolders for each organization, you can still use those custom subdomain addresses and they will all file into the "sub" folder.
• The advantage of this method is that you don't have to use rules to file the messages. But rules can be used to override the alias folder target delivery folder.

I know that's not what you asked, but it's how I deal with a large number of these subscriptions without using any rules.

Bill

[n5bb]

Originally Posted by xyzzy

I only quoted what he said. I doubt he tested it since we were just conversing back and forth through the ticket.

Well ... silly goose! This is somewhat funny ... when searching:

• fox finds fox and foxes
• foxes finds fox and foxes
• "fox" in quotes only finds fox
• "foxes" in quotes only finds foxes
• But the word "goose" has two plurals: "gooses" and "geese". So:
• goose finds goose and gooses
• gooses finds goose and gooses
• geese only finds geese
• So using goose and geese is an unfortunate incorrect example!

See stemming in:
https://www.fastmail.com/help/receive/search.html
Quotes (single ' or double ") around a word or phrase cause only those exact characters to be matched (ignoring case).

They might be using the Porter stemming algorithm. See:
https://raw.githubusercontent.com/sn...porter/voc.txt
https://tartarus.org/martin/PorterStemmer/
That stemming algorithm matches the fox/foxes and goose/gooses/geese behavior I noted above.

Bill

[xyzzy]

Originally Posted by gardenweed

When using sieve years ago, the code for this rule was one block which tested the from header and using ifany (if I recall) dumped the email into the Subscriptions folder. The block of sieve was quite long - ie many subscriptions - but it was all together in one block. Nice and neat.

When the new Rules came along (not the new new rules) this block of sieve got converted to a bucket load of individual rule tests. Looked messy to me but worked. So many rules for one action - file into Subscriptions folder.

Now, with the new-new rules, we can have multiple tests again, ie "If any"

Not sure what you are referring to. If you had existing "old" rule or set up a new rule in no preview mode then it's going to generate the sieve test code something like (ignoring the boilerplate stuff it adds now),

if anyof(address :is "from" "email address 1",
         address :is "from" "email address 2.
         etc.) {...}

It would be nicer if it generated,

address :is "from" [ "email address 1", "email address 2", "email address 3"] {...}

but it has never done that (at least in the time I've been using FM - only a few years).

If you created the rule in the new preview mode the test part looks like (still ignoring the extra boilerplate),

jmapquery text:
  {
    "conditions" : [
      { 
        "from" : "email address 1"
      },
      {
        "from" : "email address 2"
      }, 
      {
        "from" : "email address 3"
      },
    ],
    "operator" : "OR",
 }
.

All those braces on separate lines really spread this stuff out.

For my rule that puts email into my Subscriptions bucket - is it better to have many individual rules, or to have one rule that lists all the email from-addresses that should direct to my Subscriptions bucket? Pros/Cons?

IMO from your point of view for execution it probably doesn't matter since this stuff is happening as the server receives the message. It's sort of the same question I had when I saw that for the normal filter case it fileinto :copy's the message into an mailbox and later discards the original.

IMO from an aesthetic/readability point of view it's probably better to keep them together rather than creating a separate rule for each. Personally if it's a bunch all filtering into the same mailbox I probably would just use the most compact form directly in sieve and format it something like

if address :is "from" [
  "email address 1", 
  "email address 2",
  "email address 3"] {
  ...
}

Makes it easier then to see the entire list and update it as necessary. Of course that just me! I don't mind adding stuff to sieve if it's more compact than spraying it over a bunch of rules in the rules UI.

[xyzzy]

Originally Posted by n5bb

So using goose and geese is an unfortunate incorrect example!

I agree. Glad you pointed out that reference. It's still another thing I learned about FM today. Thanks.

[gardenweed]

@xyzzy

Very much like the 1st example

Sieve used to look like:

if anyof(
 header :contains "subject" "some repeating subject",
 header :contains "from" "@abcd.com.au",
 header :contains "from" "efgh.com.au",
 header :contains "from" "@stufflinfo.com",
 ....
 header :contains "from" "news@somepub.com"
 ) {
   fileinto "INBOX.Subscriptions";
  stop;
  }

Fairly neat & tidy.
Just one block of conditions for all

[xyzzy]

Originally Posted by gardenweed

Fairly neat & tidy.
Just one block of conditions for all

So if that was the original "old" rules code or you re-created it in the "new" rules in no preview mode the test portion of the code should be the same sequence of code and surrounded by the new rules "bolierplate" I omitted in my earlier example. For your example you should be seeing,

if  allof( not string :is "${stop}" "Y",
     anyof(
     header :contains "Subject" "some repeating subject",
     address :contains "From" "@abcd.com.au",
     address :contains "From" "efgh.com.au",
     address :contains "From" "@stufflinfo.com",
    ...
    address :contains "From" "news@somepub.com"
     )
   )
{ 
  set "L0_test" "Y";
  
  set "skipinbox" "Y";
  
  set "stop" "Y";
}

The rule's actions are delayed to allow certain rules to be executed in combination with other rules; to control sending to trash or spam, or to do the associated fileinto. The state switches (variables), i.e., stop, deletetotrash, spam, and hasmailbox control the order of the various operations.

I don't mind that they through a bucket of switches at this to control the various rules combinations. I just wish they didn't put a blank line after each setting which needlessly spreads the code out more than it has to.

-------

Off topic meta question: does anybody now why surrounding example code with the code formatting operation inserts so many blank lines at the end of the code? They are not there when I create or edit the post.

[gardenweed]

Originally Posted by n5bb

Since you asked a philosophical question, I will respond with my philosophy on this issue......

Thanks Bill.
Looks like a smart and organised way to do things and addresses a number of issues.

[ao1]

Is there a risk that upgrading to the new system will mess up the custom sieve rules?

[gardenweed]

Originally Posted by ao1

Is there a risk that upgrading to the new system will mess up the custom sieve rules?

Kind of.
I had to remove my custom sieve before I was allowed to upgrade.
So I converted the few custom rules I still had to the existing rules system and then removed the custom sieve.
Then I was able to select the upgrade button.

[xyzzy]

Originally Posted by ao1

Is there a risk that upgrading to the new system will mess up the custom sieve rules?

Read my post 5 in this thread.

So IMO essentially the new FM-generated sieve code should not screw up your existing additions at the beginning and/or end. But you'll have to figure out how to convert anything you had in the "old" middle two blocks to work with the new code.